Multi-armed Bandit Problems with History

In a multi-armed bandit problem, at each time step, an algorithm chooses one of the possible arms and observes its rewards. The goal is to maximize the sum of rewards over all time steps (or to minimize the regret). In the conventional formulation of the problem, the algorithm has no prior knowledge about the arms. Many applications, however, provide some data about the arms even before the algorithm starts. For example: a search engine company obtained data from a small sample of paid users on its newly developed retrieval functions. The availability of such historic data leads to the question of how online learning algorithms can best use it to reduce regret. This problem is meaningful only for the case of stochastic arms [1]. We propose algorithms which show that a logarithmic amount of historic data allows them to achieve constant regret. The work by [2] assumes that historic data collected via some policy is available to evaluate a mapping from side information to arms. In the absence of side-information, their policy evaluation strategy reduces to choosing the arm with the highest mean reward on the historic data. In a K armed stochastic bandit problem, random variable Xj,t ∈ [0, 1] (1 ≤ j ≤ K, t ≥ 1) denotes the reward incurred when the j arm is pulled the t time. For arm j, the rewards Xj,t are iid with mean μj and variance σ 2 j . The best arm is denoted by j i.e., μj∗ := max1≤i≤K μi. Historic data is denoted by X h j,t ∈ [0, 1] for 1 ≤ j ≤ K and 1 ≤ t ≤ Hj . The historic rewards for each arm are assumed to be drawn iid as well. Tj(n) denotes the number of times the arm j is pulled between times 1 and n (this excludes the pulls of the arm in the historic data). The regret at time n is defined as REG(n) := μj∗n − μj ∑K j=1 E[Tj(n)]. The per-round regret at time n is defined as REG(n)/n. Mean reward of arm j during the execution of the algorithms until its n pull is defined as X̄j,n. Analogously, the joint mean reward of arm j incorporating both the historic and the online data is X̄ j,n.